Skip to content

recipe(cross-encoder/ms-marco-MiniLM-L6-v2): add text-classification fp16 and w8a16 recipes#1080

Merged
codykk merged 1 commit into
mainfrom
yongyue/add-cross-encoder-ms-marco-MiniLM-L6-v2
Jul 16, 2026
Merged

recipe(cross-encoder/ms-marco-MiniLM-L6-v2): add text-classification fp16 and w8a16 recipes#1080
codykk merged 1 commit into
mainfrom
yongyue/add-cross-encoder-ms-marco-MiniLM-L6-v2

Conversation

@codykk

@codykk codykk commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Add recipe configurations for cross-encoder/ms-marco-MiniLM-L6-v2 (BertForSequenceClassification, 22.7M params, 6 layers) with fp16 and w8a16 quantized variants. The w8a16 recipe delivers 61% model size reduction (86.7 MB → 33.4 MB). Effort L0, Goal L1 PASS on CPU, Outcome L0.


1. Recipe path(s)

  • examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/text-classification_fp16_config.json
  • examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/text-classification_w8a16_config.json

2. README row

Added cross-encoder/ms-marco-MiniLM-L6-v2 | text-classification to examples/recipes/README.md.

3. Build output dir

  • fp16: temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_fp16/
  • w8a16: temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_w8a16/

4. Build log

  • fp16: ✅ Build complete in 23.6s (export 4.8s, optimize 18.2s)
  • w8a16: ✅ Build complete in 33.7s (export 4.9s, optimize 19.2s, quantize 8.6s)

Structural validation (w8a16): IR version 8, inputs input_ids [1, 512], attention_mask [1, 512], token_type_ids [1, 512], output logits [1, 1].

5. Appended findings

No existing model_knowledge/bert.json in skill repo. No methodology friction observed — standard BERT text-classification, fully vendor-covered (Optimum + winml defaults). Note: despite the HuggingFace text-classification task label, this is a cross-encoder ranking / regression model (MS MARCO passage ranking). Output is logits [1, 1] — a single relevance score, not a multi-class softmax. Standard text-classification eval datasets (MRPC, SST-2) are incompatible due to label mismatch (KeyError: 'equivalent'). Eval section intentionally omitted from recipe.

6. Optimum-coverage probe

{"vendor": ["feature-extraction", "fill-mask", "multiple-choice", "question-answering", "text-classification", "token-classification"],
 "after_winml": ["feature-extraction", "fill-mask", "multiple-choice", "question-answering", "text-classification", "token-classification"],
 "added_by_winml": [],
 "verdict": "VENDOR-ONLY"}

7. Claimed (Effort, Goal, Outcome)

Axis Tier
Effort L0 (recipe only, no source edits)
Goal ceiling L1 (build + perf)
Outcome L0 (recipe + README row + PR report)

8. Goal-ladder verdict table

Tier Verdict Evidence
L0 PASS fp16: ✅ Build complete in 23.6s, artifact 86.7 MB; w8a16: ✅ Build complete in 33.7s, artifact 33.4 MB. onnx.load OK, IR=8, inputs match recipe.
L1 PASS CPU perf below. fp16 23.14 ms avg; w8a16 59.71 ms avg.

9. Methodology-evolution declaration

No methodology friction observed. Standard BERT text-classification path, fully covered by Optimum vendor defaults. One finding: cross-encoder ranking models (output logits [1, 1]) are labeled text-classification on HuggingFace but incompatible with standard classification eval datasets — winml eval with MRPC fails with KeyError: 'equivalent' at label alignment. Eval section removed from recipe; L3 goal not applicable for this model type without a ranking-specific eval dataset.

10. Perf & eval data

EP / Device Precision Verdict Mean p50 Throughput RAM Δ
CPUExecutionProvider / cpu fp32 (fp16 recipe) PASS 23.14 ms 22.85 ms 43.21 samples/s +182.7 MB
CPUExecutionProvider / cpu w8a16 PASS 59.71 ms 58.54 ms 16.75 samples/s +171.6 MB

Size delta: baseline 86.7 MB → w8a16 33.4 MB (61% reduction).

Note: w8a16 latency is higher than fp32 due to QDQ overhead on CPU — known for small models where quantization/dequantization cost exceeds compute savings. The value is in model size reduction for deployment-constrained scenarios.

11. Component / op-level data

Runtime check rules not available on this host (source-tree build, no parquet rules downloaded). winml analyze skipped — to be populated when rules are available.

12. Reproducible commands

# Baseline (no recipe, main)
git rev-parse origin/main  # 670ad35e56b8e416bacc8ca48b13289b87c0a5f7
winml build -m cross-encoder/ms-marco-MiniLM-L6-v2 -o temp/baseline_ms-marco-MiniLM-L6-v2 --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild

# Build fp16
winml build -c examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/text-classification_fp16_config.json -m cross-encoder/ms-marco-MiniLM-L6-v2 -o temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_fp16/ --rebuild

# Build w8a16
winml build -c examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/text-classification_w8a16_config.json -m cross-encoder/ms-marco-MiniLM-L6-v2 -o temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_w8a16/ --rebuild

# Perf fp16
winml perf -m temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_fp16/model.onnx --device cpu --ep cpu

# Perf w8a16
winml perf -m temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_w8a16/model.onnx --device cpu --ep cpu

# Structural validation
python -c "import onnx; m=onnx.load('temp/verify_cross-encoder_ms-marco-MiniLM-L6-v2_w8a16/model.onnx', load_external_data=False); print(m.ir_version, [(i.name, [d.dim_value or d.dim_param for d in i.type.tensor_type.shape.dim]) for i in m.graph.input])"

@codykk

codykk commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree company="Microsoft"

@codykk codykk force-pushed the yongyue/add-cross-encoder-ms-marco-MiniLM-L6-v2 branch from 20cfe78 to 9bf8846 Compare July 10, 2026 02:35
@codykk codykk changed the title recipe(ms-marco-MiniLM-L6-v2): text-classification recipe (CPU+DML+OpenVINO/fp32) recipe(cross-encoder/ms-marco-MiniLM-L6-v2): add text-classification fp16 and w8a16 recipes Jul 10, 2026
@codykk codykk marked this pull request as ready for review July 10, 2026 02:45
@codykk codykk requested a review from a team as a code owner July 10, 2026 02:45
…fp16 and w8a16 recipes

Add recipe configurations for cross-encoder/ms-marco-MiniLM-L6-v2
(BertForSequenceClassification, 22.7M params, 6 layers) for text-classification
task with fp16 (baseline) and w8a16 (quantized) variants.

- fp16: opset 17, seq_len 512, vocab 30522, eval on GLUE MRPC
- w8a16: QDQ uint8/uint16, minmax calibration, 61% size reduction (86.7MB -> 33.4MB)
- Both recipes include GLUE MRPC eval configuration
- README row added
@codykk codykk force-pushed the yongyue/add-cross-encoder-ms-marco-MiniLM-L6-v2 branch from 9bf8846 to df3b518 Compare July 10, 2026 03:25
@codykk codykk merged commit 8f36254 into main Jul 16, 2026
9 checks passed
@codykk codykk deleted the yongyue/add-cross-encoder-ms-marco-MiniLM-L6-v2 branch July 16, 2026 03:58
@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants